Developing Measures of Public Perceptions of Values in Science

Dan Hicks
Emilio Lobato
University of California, Merced
values-in-science-scale.netlify.app

Background & acknowledgment

  • I’m a philosopher of science, STSer, data scientist
    • Science, values, and policy
    • Public scientific controversies
  • Emilio is a cognitive psychologist
    • Conspiracy theories & other
      “epistemically unwarranted beliefs”
    • Relationship between prejudice and EUBs
  • This project was funded out of my faculty startup funds at UC Merced

Controversy and public understanding of science

Explaining public scientific controversies

HPSTSers often explain controversies by appeal to public views on  science, values, and policy issues

But do the public hold these views?

(As a gross generalization)

  • Historians focus on actions of scientists or merchants of doubt
  • Sociologists look at very particular case studies that don’t generalize to public at large
  • Philosophers don’t offer any empirical evidence

Two notable exceptions

  • Weisberg et al. (2020)
  • Elliott et al. (2017)
    • Does transparency about values in science reduce trust?
    • Online survey experiment
    • But don’t ask about participants’ views on, eg, VFI

Developing a Values in Science Scale (VISS)

VISS items

36 items, 3 each for 12 different topics in the
science, values, and policy literature

Aims of science
Conflicts of interest
Consensus
Fact-value distinction
Fallibilism
Inductive risk

Non-subjectivity
Pluralism
Scientism
Standpoint theory
Technocracy
Value-free ideal

Data and methods

  • Online survey platform Prolific
  • Representative sample of 980 US adults
  • VISS + demographics + replication of Elliott et al. (2017)
  • Approved by UC Merced IRB


Agreement/disagreement:
All VISS items

Agreement/disagreement:
Top & bottom 5

(strongly) agree item prompt
top 5
81% aims.1 The knowledge produced by scientific research is valuable even when it has no practical applications.
74% aims.2 A primary aim of science is to improve our understanding of threats to human health and the environment.
69% scientism.2 Everyone, whatever their worldview, should accept evidence.based scientific conclusions
69% coi.2 Special interests can always find an scientist-for-hire who will support their point of view.
66% technocracy.2 When it comes to public policy, scientific evidence is more important than public opinion.
bottom 5
15% technocracy.3 Scientists have too much influence over public policy today.
13% ir.2 Scientists are not responsible for harmful effects of the technologies they develop.
12% consensus.3 The consensus of the scientific community is based on social status and prestige rather than evidence.
5% consensus.2 Scientists rarely disagree with each other about the answers to scientific questions.
3% fallible.1 Once a scientific theory has been established, it is never changed.

Ambiguous prompts

ir.1
Standards of scientific evidence are different in different situations.


Does this mean

  • legitimate standards of evidence can be appropriately different in different situations (eg, because of different downstream non-epistemic consequences); or
  • scientists illegitimately use different standards of evidence in different situations (eg, to reach their preferred conclusion)?


  • Measurement error:
    How many participants understood the prompt one way vs. the other?
  • Construct validity:
    Does the item measure the thing you intended to measure?
  • Requires understanding how participants interpret prompts

Replication of Elliott et al. (2017)

Dr. Riley Spence and BPA

Slide attributed to (fictional) Dr. Riley Spence

Example of a stimulus from Elliott et al. (2017). Slide reads: "My conclusion. Protecting public health should be a top national priority. I examined the scientific evidence on potential health risks of BPA. I conclude that BPA in consumer products is not causing harm to people."

  • 6 different slides
    • No values disclosure; protecting public health; promoting economic growth
    • BPA is/is not causing harm
  • Muenster Epistemic Trustworthiness Inventory (Hendriks, Kienhues, and Bromme 2015)
    • eg, competent-incompetent, 1-7 scale

Elliott et al. major findings

Transparency penalty
If the scientist discloses values, participants tend to perceive them as less trustworthy
Shared values effect
If the participant and scientist share the same values, participants tend to perceive the scientist as more trustworthy

Transparency penalty?

Dotplot of trust ratings for Elliott et al. (2017) and our data, broken out by whether or not the scientist disclosed values. Elliott et al. found an estimated effect of transparency of about -0.5 on a 1-7 scale; our data shows a negligible difference.

Shared values effect?

Dotplot of trust ratings for our data, broken out by participant values (economic growth or public health) and whether they share disclosed values with the scientist. The effect of shared values appears to be positive or negative depending on the participants' values.

Scientist values effect

Dotplot of trust ratings for our data, broken out by participant values (economic growth or public health) and the scientists' disclosed values. Scientist valuing public health appears to increase trust, about 0.7 on a 1-7 scale, for both public health and economic growth participants.

Future work

Future work

  • Qualitative designs to revise prompts
    • Paraphrase the prompt; think aloud
    • Likely will reduce measurement error
  • Other endpoints
    • Dueling expert scenarios
    • Trust in science-policy institutions (EPA) rather than individual scientists (Brown 2022; Biddle yesterday)
    • Policy support

Fin

values-in-science-scale.netlify.app

Extra slides

Prompts

aims-1
The knowledge produced by scientific research is valuable even when it has no practical applications.
aims-2
A primary aim of science is to improve our understanding of threats to human health and the environment.
aims-3
A primary aim of science is to develop new technology and stimulate economic growth.
coi-1
Scientists will report conclusions that they think will get them more grant money even if the data does not fully support that conclusion.
coi-2
Special interests can always find an scientist-for-hire who will support their point of view.
coi-3
Research carried out by private companies is less trustworthy than research carried out by public instutitions.
consensus-1
Progress in science happens when a lone dissenter challenges popular ideas.K
consensus-2
Scientists rarely disagree with each other about the answers to scientific questions.
consensus-3
The consensus of the scientific community is based on social status and prestige rather than evidence.
factvalue-1
Moral and social values are the kinds of things that can be confirmed or challenged by scientific evidence.
factvalue-2
If scientific findings conflict with important values that we hold, it’s appropriate to be skeptical of the science.
factvalue-3
Someone’s moral and social values are more like their taste in music than scientific facts.
fallible-1
Once a scientific theory has been established, it is never changed.K
fallible-2
When a scientific theory changes or is revised, it means that the research that went into it initially was flawed.
fallible-3
Scientific findings can be good enough to act on even when they’re not entirely certain.
ir-1
Standards of scientific evidence are different in different situations.
ir-2
Scientists are not responsible for harmful effects of the technologies they develop.
ir-3
When scientific research has potential consequences for society, scientists must take these consequences into account when drawing conclusions.
nonsubj-1
The results of scientific studies are always ambiguous and require interpretation.
nonsubj-2
Good scientific research is always free of assumptions and speculation.
nonsubj-3
Scientific conclusions are never a matter of interpretation or judgment.
pluralism-1
Scientific investigations always require laboratory experiments.K
pluralism-2
Scientific research is always conducted in the following order: 1. Observation, 2. Hypothesis, 3. Experiment, 4. ConclusionK
pluralism-3
All scientists use the same strict requirements for determining when empirical data confirm a tested hypothesis.
scientism-1
Science is more reliable than any other way of producing knowledge.
scientism-2
Everyone, whatever their worldview, should accept evidence-based scientific conclusions
scientism-3
Science is the only source of justified belief or knowledge about ourselves and the world.
stdpt-1
A scientific understanding of the social word depends on the background of the scientists carrying out the research.
stdpt-2
People on the margins of society often have a unique understanding of that society that can benefit scientific research.
stdpt-3
Science has a long history of ignoring the unique insights of women and people of color.
technocracy-1
Public policy should wait until all relevant scientific questions have been settled.
technocracy-2
When it comes to public policy, scientific evidence is more important than public opinion.
technocrac-3
Scientists have too much influence over public policy today.
vfi-1
Scientists are more objective than other people.K
vfi-2
Scientists do not use imagination or creativity because doing so interferes with objectivity.K
vfi-3
The evaluation and acceptance of scientific results must not be influenced by social and ethical values.K

Factor analysis results

VISS factors

Screenshot of the factor correlation matrix. The factors are labeled scientism, VIS (values in science), cynicism, power, textbook (science), and VFI (value-free ideal)

⚠️ Psychometric properties of these factors aren’t great ⚠️

Factor analysis fit indices

How well does the fitted FA model approximate the observed covariance structure? (Hu and Bentler 1998)


statistic range want our model
Comparative fit index (CFI) 0-1 large 0.6
Adjusted-goodness-of-fit index (AGFI) 0-1 large 0.8
Root-mean-square error of approximation (RMSEA) >0 small 0.07
Standardized root-mean-square residual (SRMSR) >0 small 0.09

Scientism

Density plot for scientism factor scores. The mode is just below 3.5, and values range from 1 to 5.

  • Science is more reliable than any other way of producing knowledge.
  • Scientific findings can be good enough to act on even when they’re not entirely certain.
  • The knowledge produced by scientific research is valuable even when it has no practical applications.

Plot of scientism against political ideology. There's a weak negative association: on average conservatives are about 0.5 point less scientistic than liberals.

  • Scientists are not responsible for harmful effects of the technologies they develop.
  • When it comes to public policy, scientific evidence is more important than public opinion.
  • Moral and social values are the kinds of things that can be confirmed or challenged by scientific evidence.

Cynicism

Density plot for cynicism factor scores. The mode is about 2.5, and values range from 1 to about 4.5.

  • Scientists will report conclusions that they think will get them more grant money even if the data does not fully support that conclusion.K
  • The consensus of the scientific community is based on social status and prestige rather than evidence.
  • If scientific findings conflict with important values that we hold, it’s appropriate to be skeptical of the science.
  • The results of scientific studies are always ambiguous and require interpretation.

Plot of cynicism against political ideology. There's a positive association: on average conservatives are about 1 point more cynical than liberals.

  • When a scientific theory changes or is revised, it means that the research that went into it initially was flawed.
  • Special interests can always find an scientist-for-hire who will support their point of view.
  • Standards of scientific evidence are different in different situations.
  • A scientific understanding of the social world depends on the background of the scientists carrying out the research.

Factors and trust

Estimated effects plot for all six VISS factors. Power, VFI, and VIS factors are essentially flat. Scientism and textbook are associated with increased trust, while cynicism is associated with decreased trust. However, all effects are modest, highly unlikely to be greater than +/- 1 on the 1-7 trust scale.

References

Brown, Matthew. 2022. “Trust, Expertise and Scientific Authority in Democracy.” Michigan State University, February 20. https://www.youtube.com/watch?v=y3XeP6e646g.
Cook, John, Dana Nuccitelli, Sarah A. Green, Mark Richardson, Bärbel Winkler, Rob Painting, Robert Way, Peter Jacobs, and Andrew Skuce. 2013. “Quantifying the Consensus on Anthropogenic Global Warming in the Scientific Literature.” Environmental Research Letters 8 (2): 024024. https://doi.org/10.1088/1748-9326/8/2/024024.
Elliott, Kevin C., Aaron M. McCright, Summer Allen, and Thomas Dietz. 2017. “Values in Environmental Research: Citizens’ Views of Scientists Who Acknowledge Values.” PLOS ONE 12 (10): e0186049. https://doi.org/10.1371/journal.pone.0186049.
Elliott, Kevin C., and David B. Resnik. 2014. “Science, Policy, and the Transparency of Values.” Environmental Health Perspectives, March. https://doi.org/10.1289/ehp.1408107.
Fernandez Pinto, Manuela. 2020. “Commercial Interests and the Erosion of Trust in Science.” Philosophy of Science, July. https://doi.org/10.1086/710521.
Fernández Pinto, Manuela, and Daniel J. Hicks. 2019. “Legitimizing Values in Regulatory Science.” Environmental Health Perspectives 127 (3): 035001. https://doi.org/10.1289/EHP3317.
Goldenberg, Maya J. 2021. Vaccine Hesitancy: Public Trust, Expertise, and the War on Science. University of Pittsburgh Press.
Grasswick, Heidi. 2018. “Understanding Epistemic Trust Injustices and Their Harms.” Royal Institute of Philosophy Supplements 84 (November): 69–91. https://doi.org/10.1017/S1358246118000553.
Hendriks, Friederike, Dorothe Kienhues, and Rainer Bromme. 2015. “Measuring Laypeople’s Trust in Experts in a Digital Age: The Muenster Epistemic Trustworthiness Inventory (METI).” PLOS ONE 10 (10): e0139309. https://doi.org/10.1371/journal.pone.0139309.
Hu, Li-tze, and Peter M. Bentler. 1998. “Fit Indices in Covariance Structure Modeling: Sensitivity to Underparameterized Model Misspecification.” Psychological Methods 3 (4): 424–53. https://doi.org/10.1037/1082-989X.3.4.424.
John, Stephen. 2017. “Epistemic Trust and the Ethics of Science Communication: Against Transparency, Openness, Sincerity and Honesty.” Social Epistemology 32 (2): 75–87. https://doi.org/10.1080/02691728.2017.1410864.
Jordan, Catherine, Susan Gust, and Naomi Scheman. 2011. “The Trustworthiness of Research: The Paradigm of Community-Based Research.” In Shifting Ground: Knowledge and Reality, Transgression and Trustworthiness, by Naomi Scheman, 170–90. Oxford University Press. http://books.google.ca/books?id=ZvqLRu30ipsC&pg=PA149&dq=intitle:Shifting+Ground+inauthor:scheman&hl=&cd=1&source=gbs_api.
Kovaka, Karen. 2021. “Climate Change Denial and Beliefs about Science.” Synthese 198 (3): 2355–74. https://doi.org/10.1007/s11229-019-02210-z.
Largent, Mark A. 2012. Vaccine: The Debate in Modern America. JHU Press.
Lombrozo, Tania, Anastasia Thanukos, and Michael Weisberg. 2008. “The Importance of Understanding the Nature of Science for Accepting Evolution.” Evolution: Education and Outreach 1 (3): 290–98. https://doi.org/10.1007/s12052-008-0061-8.
McHugh, Nancy Arden. 2011. “More Than Skin Deep: Situated Communities and Agent Orange in the Aluoi Valley, Vietnam.” In Feminist Epistemology and Philosophy of Science, edited by Heidi E Grasswick, 183–204. Power In Knowledge. Springer. http://books.google.com/books?id=fsLVt65PxrAC&printsec=frontcover&dq=intitle:Feminist+Epistemology+and+Philosophy+of+Science&hl=&cd=1&source=gbs_api.
Myers, Krista F., Peter T. Doran, John Cook, John E. Kotcher, and Teresa A. Myers. 2021. “Consensus Revisited: Quantifying Scientific Agreement on Climate Change and Climate Expertise Among Earth Scientists 10 Years Later.” Environmental Research Letters 16 (10): 104030. https://doi.org/10.1088/1748-9326/ac2774.
Navin, Mark. 2015. Values and Vaccine Refusal: Hard Questions in Ethics, Epistemology, and Health Care. Routledge.
Nelson, Alondra. 2011. Body and Soul: The Black Panther Party and the Fight Against Medical Discrimination. U of Minnesota Press.
Oreskes, N. 2004. “The Scientific Consensus on Climate Change.” Science 306 (5702): 1686. https://doi.org/10.1126/science.1103618.
Scheman, Naomi. 2001. “Epistemology Resuscitated: Objectivity as Trustworthiness.” Engendering Rationalities, 23–52. http://books.google.com/books?hl=en&lr=&id=CLl5HMhRWBsC&oi=fnd&pg=PA23&dq=epistemology+resuscitated+naomi+scheman&ots=2RBRcsu4dg&sig=h7xRsp8flRZFpsFW3rCfCw9R7C4.
Washington, Harriet A. 2008. Medical Apartheid: The Dark History of Medical Experimentation on Black Americans from Colonial Times to the Present. Knopf Doubleday Publishing Group.
Weisberg, Deena Skolnick, Asheley R. Landrum, Jesse Hamilton, and Michael Weisberg. 2020. “Knowledge about the Nature of Science Increases Public Acceptance of Science Regardless of Identity Factors.” Public Understanding of Science, December, 0963662520977700. https://doi.org/10.1177/0963662520977700.
Wynne, Brian. 1989. “Sheepfarming After Chernobyl: A Case Study in Communicating Scientific Information.” Environment: Science and Policy for Sustainable Development 31 (2): 10–39. https://doi.org/10.1080/00139157.1989.9928930.